Reinforcement Learning with Partially Known World Dynamics
نویسنده
چکیده
Reinforcement learning would enjoy better success on real-world problems if domain knowledge could be imparted to the algorithm by the modelers. Most problems have both hidden state and unknown dynamics. Partially observable Markov decision processes (POMDPs) allow for the modeling of both. Unfortunately, they do not provide a natural framework in which to specify knowledge about the domain dynamics. The designer must either admit to knowing nothing about the dynamics or completely specify the dynamics (thereby turning it into a planning problem). We propose a new framework called a partially known Markov decision process (PKMDP) which allows the designer to specify known dynamics while still leaving portions of the environment’s dynamics unknown. The model represents not only the environment dynamics but also the agent’s knowledge of the dynamics. We present a reinforcement learning algorithm for this model based on importance sampling. The algorithm incorporates planning based on the known dynamics and learning about the unknown dynamics. Our results clearly demonstrate the ability to add domain knowledge and the resulting benefits for learning.
منابع مشابه
Solving Partially Observable Reinforcement Learning Problems with Recurrent Neural Networks
In partially observable environments effective reinforcement learning (RL) is still a fairly open question. Most common algorithms fail to produce good results for those problems. However, many real-world applications are characterized by those difficult environments. In this paper we propose the application of recurrent neural networks (RNN) to identify in a first step the complete state space...
متن کاملManifold Embeddings for Model-Based Reinforcement Learning of Neurostimulation Policies
Real-world reinforcement learning problems often exhibit nonlinear, continuous-valued, noisy, partially-observable state-spaces that are prohibitively expensive to explore. The formal reinforcement learning framework, unfortunately, has not been successfully demonstrated in a real-world domain having all of these constraints. We approach this domain with a two-part solution. First, we overcome ...
متن کاملManifold Embeddings for Model-Based Reinforcement Learning under Partial Observability
Interesting real-world datasets often exhibit nonlinear, noisy, continuous-valued states that are unexplorable, are poorly described by first principles, and are only partially observable. If partial observability can be overcome, these constraints suggest the use of model-based reinforcement learning. We experiment with manifold embeddings to reconstruct the observable state-space in the conte...
متن کاملModel-Assisted Approaches for Relational Reinforcement Learning: Some challenges for the SRL community
For a relational reinforcement learning (RRL) agent, learning a model of the world can be very helpful. However, in many situations learning a perfect model is not possible. Therefore, only probabilistic methods capable of taking uncertainty into account can be used to exploit the collected knowledge. It is clear then that RRL offers an interesting testbed for statistical relational learning me...
متن کاملReinforcement Learning for Decentralized Planning Under Uncertainty (Doctoral Consortium)
Decentralized partially-observable Markov decision processes (Dec-POMDPs) are a powerful tool for modeling multi-agent planning and decision-making under uncertainty. Prevalent Dec-POMDP solution techniques require centralized computation given full knowledge of the underlying model. But in real world scenarios, model parameters may not be known a priori, or may be difficult to specify. We prop...
متن کامل